End-to-End Evaluation of Machine Interpretation Systems: A Graphical Evaluation Tool

VERBMOBIL as a long-term project of the Federal Ministry of Education, Science, Research and Technology aims at developing a mobile translation system for spontaneous speech. The source-language input consists of human speech (English, German or Japanese), the translation (bidirectional English-German and Japanese-German) and target-language output is effected by the VERBMOBIL system. As to the innovative character of the project new methods for end-to-end evaluation had to be developed by a subproject which has been established especially for this purpose. In this paper we present criteria for the evaluation of speech-tospeech translation systems and a tool for judging the translation quality which is called Graphical Evaluation Tool (GET). 1 This work was funded by the Federal Ministry of Education, Science, Research and Technology (BMBF) in the framework of the VERBMOBIL Project under Grant 01 IV 101 A/O and in the framework of the SFB 538 Mehrsprachigkeit (Collaborative Research Center No. 538 Multilingualism) by the Deutsche Forschungsgemeinschaft (DFG). The responsibility for the contents of this study lies with the authors. 2 To simplify the presentation of this paper, we only refer to the language pair German-English. Introduction The performance of evaluation very often is driven by the characteristics of the system that has to be judged (Andenfilger, 1994). As to the Verbmobil project the evaluation should meet three aspects: • the needs of the developers, • the needs of the user, • the constraints on the evaluation of translation quality in general. In our concept and performance of evaluation we tried to combine these three aspects but one should keep in mind that at least the constraints on translation quality in general were meant to describe human translation with all its varieties and specific stylistic features. As to the special case of machine interpretation still only texts from limited domains can be transferred. So in our view it seems to be legitimate to simplify some of the procedures that are applied to the evaluation of human translation. An evaluation method based on any well known standard (EAGLES, 1995; Spark Jones and Galliers, 1996; Manzi, 1996) could not have integrated the three cited aspects, as traditional evaluation methods are intended for comparative evaluations more than for the investigation of a system during its development; therefore, to meet the requirements we had, we developed an integrated methodology and a tool for speech-to-speech quality evaluation which also allows easy access to the data.